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About  the  Technical  Note  Series  on  Architecture  Evaluation  in 


the  Department  of  Defense 


The  Product  Line  Systems  Program  is  publishing  a  series  of  technical  notes  designed  to 
condense  knowledge  about  architecture  evaluation  practices  into  a  concise  and  usable  form 
for  the  Department  of  Defense  (DoD)  acquisition  manager  and  practitioner.  This  series  is  a 
companion  to  the  Software  Engineering  Institute  (SEI)  series  on  product  line  acquisition  and 
business  practices  [Bergey  99]. 

Each  technical  note  in  the  series  will  focus  on  the  use  of  architecture  evaluation  and,  in 
particular,  on  applying  the  SEI’s  architecture  tradeoff  analysis  technology  in  the  DoD.  Our 
objective  is  to  provide  practical  guidance  on  ways  to  integrate  sound  architecture  evaluation 
practices  into  their  acquisitions.  This  series  of  technical  notes  will  lay  down  a  conceptual 
foundation  for  DoD  architecture  evaluation  practice. 
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Abstract 


Software  architecture  is  critical  to  the  quality  of  a  software -intensive  system.  For  an 
acquisition  organization,  such  as  the  Department  of  Defense  (DoD),  the  ability  to  evaluate 
software  architectures  early  in  an  acquisition  can  have  a  favorable  impact  on  the  delivered 
system.  This  technical  note  discusses  the  role  of  software  architecture  evaluations  in  a  system 
acquisition  and  describes  the  contractual  elements  that  are  needed  to  accommodate 
architecture  evaluations  in  an  acquisition.  The  note  then  provides  an  example  of  contractual 
language  that  incorporates  the  Architecture  Tradeoff  Analysis  Method™  (ATAMsm)  as  a 
software  architecture  evaluation  method  in  a  system  acquisition. 


Architecture  Tradeoff  Analysis  Method  and  ATAM  are  service  marks  of  Carnegie  Mellon  University. 
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1  Introduction 


The  software  architecture  of  a  system  significantly  influences  the  overall  functionality, 
performance,  and  quality  of  that  system.  The  use  of  software  architecture  evaluations  early  in 
a  system  acquisition  can  help  mitigate  many  of  the  technical  risks  associated  with  system 
development,  thereby  improving  the  ability  of  the  acquisition  to  achieve  the  stated  system 
objectives  [Fisher  98].  In  an  acquisition  context,  these  evaluations 

•  provide  a  proactive  means  of  gaining  early  visibility  into  critical  design  decisions  that 
will  drive  the  entire  system-development  effort 

•  can  be  performed  before  a  system  is  built  (e.g.,  during  engineering  design  processes)  to 
determine  if  the  system  will  satisfy  its  desired  qualities 


This  technical  note  discusses  where  the  Architecture  Tradeoff  Analysis  MethodSM  (ATAMsm) 
or  other  architecture  evaluation  methods  can  be  employed  most  advantageously  in  a  system 
acquisition.  It  also  reviews  the  steps  of  the  ATAM  and  provides  sample  acquisition  language1 
(i.e.,  contractual  wording  for  a  statement  of  work  [SOW]  and  system  specification)  that  will 
enable  an  acquirer  to  apply  an  architecture  evaluation  method  such  as  the  ATAM,  during  the 
post-award  phases  of  an  acquisition.2 


Architecture  Tradeoff  Analysis  Method  and  ATAM  are  service  marks  of  Carnegie  Mellon  University. 

1  Every  acquisition  is  considered  unique.  The  acquisition  language  provided  in  this  technical  note  should  not 
be  applied  directly  to  all  acquisitions.  It  is  strongly  recommended  that  the  language  be  adapted  by  the 
acquirer  to  his/her  specific  needs. 

2  Future  technical  notes  will  provide  language  for  using  the  ATAM  in  other  acquisition  phases  and  with  other 
system  acquisition  strategies. 
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2  Software  Architecture  Evaluation  in  System  Acquisitions 


In  this  technical  note,  we  consider  the  activities  corresponding  to  three  phases  of  an 
acquisition:  pre-award,  award,  and  post-award  [Bergey  99].  These  activities  are  illustrated  in 
Figure  1.  Software  architecture  evaluation  can  potentially  play  a  role  in  the  award  and  post¬ 
award  phases  to  help  lower  the  risks  associated  with  an  acquisition. 


During  the  award  phase,  architecture  evaluations  can  be  used  to  evaluate  suppliers’  overall 
approaches  to  system  design,  to  assess  the  strengths  and  weaknesses  of  competing 
architectures,  and  to  identify  risks  to  the  program. 


After  contract  award,  software  architecture  evaluations  can  be  used  for  contract  management, 
by  enabling  acquirers  to  evaluate  both  supplier  and  product  performance. 


Initiation 
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Completion 
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RFP 

Proposals 

TIMs 

Deliverables 

A _ A _ 

_ A_ 

_ A _ A _ 

-A _ 
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•  RFP  Preparation 

*  Contract  administration 

*  Solicitation 

•  Contract  performance 

*  Contract  technical  monitoring 

•  Proposal  evaluation 

•  Best  and  final  offers  (BAFO) 

•  Source  selection 

Figure  1:  Phases  of  an  Acquisition 


To  use  software  architecture  evaluation  in  either  the  award  phase  (e.g.,  source  selections)  or 
the  post-award  phase  (e.g.,  contract  management),  the  solicitation  package  must  contain  the 
criteria  for  proposal  and  product  evaluation  and  include  the  architecture  evaluation  method  to 
be  used  as  part  of  the  architecture  requirements. 
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2.1  Pre-Award  and  Award  Phase  for  a  System-Development  Contract 

Acquisition  planning  precedes  the  entire  solicitation  process  and  includes  generating  and 
validating  product  requirements  (e.g.,  functional  and  quality  requirements  such  as  reliability 
or  performance). 

In  the  pre-award  phase,  a  solicitation  package  is  developed.  It  tells  potential  suppliers  what 
the  requirements3  of  the  acquisition  are,  how  to  prepare  their  proposals,  how  proposals  will 
be  evaluated,  and  when  to  submit  their  proposals  [Cooper  99].  Solicitation  packages  take 
various  forms  and  are  referred  to  differently.  However,  they  all  have  the  same  characteristics 
noted  here.  We  will  use  the  common  term  “request  for  proposals”  (RFP)  to  refer  to 
solicitation  packages. 

As  shown  in  Figure  2,  the  RFP  typically  contains  sections  A  through  M.  These  sections 
provide  information  that  must  be  distributed  to  potential  suppliers.  Depending  upon  the 
acquiring  organization’s  policies  and  processes,  the  sections  may  be  incorporated  in  different 
ways.  Most  RFPs,  however,  contain  the  same  type  of  information. 


Section  M:  Evaluation 
Factors 


Section  L:  Proposal 
Preparation  Instructions 


Section  J:  List  of  Contract 
Deliverables 


Section  H:  Special  Contract 
Requirements 


Section  C:  Statement  of 
Work 


Section  B:  Supplies, 
Services,  and  Prices 


Figure  2:  Contents  of  Request  for  Proposals  (RFPs) 


The  term  “requirements”  encompasses  all  requirements  of  the  acquisition,  including  product  requirements, 
where  the  term  product  may  mean  a  specific  system  or  services  [Cooper  99]. 
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The  RFP  and  eventual  contract  language  should 

•  address  the  acquisition  requirements  of  the  project 

•  comply  with  regulations,  policies,  and  other  guidance 

•  clearly  describe  product  requirements  in  terms  of  functionality,  performance,  and  quality 

•  protect  the  interests  of  both  the  acquirer  (buyer)  and  the  supplier  (contractor) 


What  goes  into  an  RFP  and  the  resulting  contract  depends  largely  upon  the  acquirer’s 
knowledge  and  objectives  for  the  acquisition.  For  our  interest,  the  RFP  sections  must  include 
the  requirement  for  a  software  architecture  evaluation.  As  a  result,  in  this  technical  note,  we 
are  interested  in  Sections  C,  L,  and  M.  We  will  discuss  these  sections  to  demonstrate  some  of 
the  considerations  needed  to  incorporate  a  software  architecture  evaluation  into  an 
acquisition. 

2.1.1  Section  C 

Section  C  contains  supplier  work  requirements  in  the  form  of  a  statement  of  objectives 
(SOO)  or  statement  of  work  (SOW)  along  with  product  requirements  such  as  a  system 
performance  specification  (containing  functional  and  quality  requirements).  If  an  architecture 
evaluation  method  is  to  be  required,  both  the  SOW  and  the  product  requirements  must 
properly  define  the  specific  method,  such  as  the  ATAM,  as  well  as  how  the  software 
architecture  evaluation  method  will  be  used  and  implemented.  This  information  must  be 
integrated  and  compatible  with  other  acquisition  requirements  that  are  part  of  the  RFP. 

Statement  of  Work  (SOW) 

The  statement  of  work  (SOW)  describes  what  the  supplier  must  accomplish.  In  terms  of  an 
evaluation  method,  the  SOW  describes  which  evaluation  steps  are  the  supplier’s 
responsibilities.  The  evaluation  steps  in  the  SOW  must  be  consistent  with  the  overall 
acquisition.  In  addition,  it  should  indicate  if  certain  evaluation  steps  are  to  be  performed 
jointly  by  the  acquirer  and  the  system  supplier. 

Product  Requirements 

A  system  specification  typically  has  two  main  sections  of  interest.  Section  1  specifies 
functional  and  quality  requirements  for  the  system.  Here,  quality  requirements  refer  to  those 
quality  attributes  of  the  system  and  their  respective  characterizations.  Modifiability, 
reliability,  and  security  are  examples  of  the  types  of  system  quality  attributes  that  may  be 
considered.  For  example,  if  reliability  is  a  required  quality  attribute,  a  characterization  might 
be  that  "the  system  will  not  fail  under  maximum  load  conditions.”  Eliciting  the  quality 
attributes  of  primary  interest  as  well  as  their  characterizations  for  the  system  in  question  are 
part  of  the  ATAM. 


4 


CMU/SEI-2001  -TN-009 


Section  2  of  the  system  specification  describes  the  software  architecture  evaluation  methods, 
such  as  the  ATAM,  to  be  used  in  determining  if  the  software  architecture  can  support  the 
satisfaction  of  the  requirements  in  Section  1 . 

2.1.2  Section  L 

Section  L  (Proposal  Preparation  Instructions)  describes  what  potential  suppliers  should 
address  in  their  proposals  and  the  response  that  is  required.  Typically,  the  acquirer  would  ask 
the  potential  suppliers  for  responses  in  several  volumes,  such  as  a  technical  volume,  past 
performance  volume,  management  volume,  and  cost  volume.  There  are  no  set  rules  for  what 
these  volumes  exactly  contain.  In  the  technical  volume,  an  acquirer  may  ask  potential 
suppliers  to  describe  their  proposed  approach  for  implementing  the  software  architecture 
requirements  and  performing  an  architecture  evaluation.  In  the  past  performance  volume,  an 
acquirer  may  ask  suppliers  to  describe  previous  work  on  software  architecture  development 
and  architecture  evaluation. 

2.1.3  Section  M 

Section  M  (Evaluation  Factors  for  Award)  tells  potential  suppliers  how  their  proposals  will  be 
evaluated.  This  typically  includes  specifying 

•  what  areas  (i.e.,  factors  and  subfactors)  of  the  supplier’s  proposed  approach  are  to  be 
evaluated  as  part  of  the  proposal  evaluation 

•  the  specific  criteria  to  be  used  forjudging  the  supplier’s  proposed  approach  to  meeting 
the  RFP/contract  requirements  for  these  factors  and  subfactors 


To  incorporate  architecture  evaluation,  Section  M  must  specify  how  the  architecture 
evaluation  will  relate  to  the  factors  and  subfactors.  And,  it  must  specify  the  criteria  to  be  used 
in  judging  the  bidder's  approach  to  satisfying  the  RFP/contract  architecture  requirements. 

From  a  contracting  officer’s  perspective,  releasing  the  RFP  defines  the  official  beginning  of 
the  solicitation.  After  the  solicitation  period  formally  closes,  source  selection  commences 
with  a  proposal  evaluation  and  ends  with  a  contract  award.  Specifying  an  architecture 
evaluation  as  part  of  the  source  selection  process  can  be  an  effective  way  to  evaluate  the  risks 
associated  with  the  proposed  software  architecture.  The  results  can  be  used  as  part  of  the 
evaluation  of  the  proposals  as  long  as  the  evaluation  criteria  are  stated  to  accommodate  these 
results. 
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2.2  Post-Award  Phase  for  a  System-Development  Contract 


Making  architecture  evaluation  a  contractual  checkpoint  is  an  effective  way  to  gain  insight 
into  the  architecture’s  ability  to  satisfy  system  requirements  early  in  its  development.  Such  an 
evaluation  can  help  the  acquirer  and  supplier 

•  select  an  architecture  from  among  several  candidate  architectures 

•  surface  and  mitigate  risks  associated  with  architectural  decisions  prior  to  development 

•  better  understand  the  ability  of  the  software  architecture  to  support  system  requirements 

•  better  understand  architectural  decisions 

•  improve  architectural  approaches 

•  evaluate  the  quality  of  evolving  products  as  required  by  contract 

•  resolve  issues  through  an  acquirer-supplier  team  approach 


It  is  important  to  note  that  what  happens  during  the  post-award  phase  critically  depends  on 
what  has  been  included  in  the  RFP  and  the  resulting  contract.  Ultimately,  the  negotiated 
contract  will  govern  what  is  permissible  during  the  contract  performance  phase  and  what 
tasks  and  products  will  be  the  supplier’s  responsibility.  Figure  3  shows  an  example  of  where 
architectures  and  their  evaluations  may  come  into  play. 


RFP 

Issued 

A _ 


Proposals  Contract 
Received  Awarded 

_ A _ A _ 


System  Development  Underway 


Pre-Award  Phase 


Award 

Phase 


Post-Award  Phase 


Offerors  Offerors 

describe  software  conduct  a  formal 

architecture  in  architecture  walkthrough 

proposal  as  part  of  their 

source  selection 
technical  presentation 


Contractual  checkpoints 
require  contractor  to  conduct  several 
in-situ  software  architecture  evaluations 

during  system  development 
in  collaboration  with 
the  acquiring  organization 


Figure  3:  Acquisition  Opportunities  for  Conducting  Architecture  Evaluations 


There  are  certainly  other  ways  to  incorporate  architecture  evaluation  in  an  acquisition 
depending  on  the  acquisition  strategy.  For  example,  an  evaluation  can  be  included  in  a  two- 
phase  acquisition  with  a  “down  select”  to  award  the  final  contract. 
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3  Architecture  Tradeoff  Analysis  Method  (AT AM) 


The  Architecture  Tradeoff  Analysis  Method  (ATAM)  is  a  technique  for  evaluating  software 
architecture.  The  technical  staff  of  the  Software  Engineering  Institute  (SEI)  developed  and 
refined  this  method  over  the  past  five  years  [Kazman  00].  The  ATAM  not  only  can  evaluate 
specific  architecture  quality  attributes,  it  allows  engineering  tradeoffs  to  be  made  among 
possibly  conflicting  system  quality  goals.  In  this  way,  the  ATAM  evaluation  can  detect  areas 
of  potential  risk  within  the  architecture  of  a  complex  software -intensive  system. 

The  Architecture  Tradeoff  Analysis  Method  has  several  advantages.  It  can  be  done  early  in 
the  software -development  life  cycle.  It  can  be  performed  quickly  and  inexpensively.  The 
method  involves  project  decision-makers,  other  stakeholders  including  managers,  developers, 
maintainers,  testers,  re-users,  end  users,  customers,  and  an  architecture  evaluation  team. 

These  groups  collaborate  to  determine  the  critical  quality  attributes  of  the  system.  The  ATAM 
provides  an  effective  means  to  evaluate  the  consequences  of  alternative  architecture  decisions 
in  light  of  specified  quality  attributes.4  The  method  ensures  that  the  right  questions  are  asked 
to  uncover 

•  risks — architecture  decisions  that  might  create  future  problems  in  some  quality  attribute 

•  sensitivity  points — properties  of  one  or  more  components  (and/or  component 
relationships)  that  are  critical  for  achieving  a  particular  quality  attribute  response  (i.e.,  a 
slight  change  in  a  property  can  make  a  significant  difference  in  a  quality  attribute) 

•  tradeoffs — decisions  affecting  more  than  one  quality  attribute 


There  are  nine  specific  steps  in  the  ATAM  evaluation  that  fall  into  four  general  types  of 
activities,  which  are  described  on  pages  7-8. 


Presentation 

Step  1:  Present  the  ATAM.  The  method  is  described  to  the  assembled  stakeholders 
(typically  customer  representatives,  the  architect  or  architecture  team,  user  representatives, 
maintainers,  administrators,  managers,  testers,  integrators,  etc.). 


The  ATAM  is  not  a  precise  mathematical  analysis.  It  is  intended  to  analyze  an  architecture  with  respect  to  its 
quality  attributes,  not  its  functional  correctness. 
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Step  2:  Present  business  drivers.  The  project  manager  describes  the  business  goals  that  are 
motivating  the  development  effort  and  hence  the  primary  architecture  drivers  (e.g.,  high 
availability,  time  to  market,  high  security,  etc.). 

Step  3:  Present  architecture.  The  architect  describes  the  proposed  architecture,  focusing  on 
how  it  addresses  the  business  drivers. 


Investigation  and  Analysis 

Step  4:  Identify  architecture  approaches.  Architecture  approaches  are  identified  by  the 
architect,  but  are  not  analyzed. 

Step  5:  Generate  quality  attribute  utility  tree.  The  quality  attributes  that  comprise  system 
“utility”  (performance,  reliability,  security,  modifiability,  etc.)  are  elicited.  These  are 
specified  down  to  the  level  of  scenarios,  annotated  with  stimuli  and  responses,  and 
prioritized. 

Step  6:  Analyze  architecture  approaches.  Based  upon  the  high-priority  factors  identified  in 
Step  5,  the  architecture  approaches  that  address  those  factors  are  elicited  and  analyzed.  For 
example,  an  architecture  approach  aimed  at  meeting  performance  goals  will  be  subjected  to  a 
performance  analysis.  During  this  step,  architecture  risks,  sensitivity  points,  and  tradeoff 
points  are  identified. 


Testing 

Step  7:  Brainstorm  and  prioritize  scenarios.  Based  upon  the  example  scenarios  generated 
in  the  utility  tree  step,  a  larger  set  of  scenarios  is  elicited  from  the  entire  group  of 
stakeholders.  This  set  of  scenarios  is  prioritized  via  a  voting  process  involving  the  entire 
stakeholder  group. 

Step  8:  Analyze  architecture  approaches.  This  step  reiterates  Step  6;  but  here,  the  highly 
ranked  scenarios  from  Step  7  are  considered  to  be  test  cases  for  architecture  approaches 
determined  thus  far.  These  test  case  scenarios  may  uncover  additional  architecture 
approaches,  risks,  sensitivity  points,  and  tradeoff  points  that  are  then  documented. 


Reporting 

Step  9:  Present  results.  Based  upon  the  information  collected  in  the  ATAM  (styles, 
scenarios,  attribute-specific  questions,  the  utility  tree,  risks,  sensitivity  points,  tradeoffs),  the 
evaluation  team  presents  its  findings  to  the  assembled  stakeholders  and  details  this 
information,  along  with  any  proposed  mitigation  strategies,  in  a  written  report. 
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The  timing  and  the  parties  responsible  for  performing  each  of  these  steps  will  depend  upon 
the  acquisition  strategy  of  the  acquirer  and  the  technical  resources  available.  It  is  important  to 
have  the  roles  and  responsibilities  understood  before  incorporating  the  ATAM  or  any 
software  evaluation  method  in  an  acquisition.  The  sections  of  the  RFP  and,  ultimately,  the 
contract  must  clearly  reflect  this  understanding. 
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4  Using  the  AT AM  in  a  System  Acquisition:  An  Example 


As  we  have  indicated,  there  are  many  ways  to  incorporate  architecture  evaluations  into  an 
acquisition.  In  this  section,  we  have  selected  one  approach  to  illustrate  applying  the  ATAM  in 
an  acquisition.  We  will  initially  describe  the  acquisition  approach  and  highlight  appropriate 
RFP  language.  Examples  of  the  language  are  given  in  the  appendices.5  It  must  be 
remembered  that  software  architecture  requirements  and  design  contribute  substantially  to  the 
achievement  of  system  requirements.  So,  the  language  in  the  appendices  must  be  viewed  as 
only  a  portion  of  the  RFP  language  and  be  integrated  into  the  overall  context  of  the  system 
acquisition.  If  a  software-only  system  is  being  acquired,  developing  the  appropriate  language 
is  much  easier;  this  is  not  the  norm,  however,  in  the  DoD  environment. 


4.1  Example  Architecture  Evaluation  Approach  for  a  System  Acquisition 

The  acquisition  approach  we  will  use  is  illustrated  in  Figure  4,  along  with  an  indication  of 
how  an  architecture  evaluation  is  related  to  the  typical  system  development  tasks  specified  in 
an  SOW.  The  approach  is  based  on  using  software  architecture  evaluations  as  contractual 
checkpoints  for  both  the  architecture  development  (initial  system  development)  and  the  full 
system  implementation  tasks.  Architects  can  use  the  results  of  the  evaluation  to  take 
corrective  action  early  in  the  development  cycle,  thereby  minimizing  or  even  avoiding  the 
cost  and  effort  of  later  rework. 


Every  acquisition  is  considered  unique.  The  acquisition  language  provided  in  this  technical  note  should  not 
be  applied  directly  to  all  acquisitions.  It  is  strongly  recommended  that  the  language  be  tailored  to  the 
acquirer’s  specific  needs. 
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Figure  4:  Integration  of  Architecture  Evaluation  with  System  Development 
Tasks  and  Evaluations 


1)  In  this  example,  the  acquirer  is  expected  to  include  preliminary  business  drivers  as  part  of 
the  product  requirements  of  the  RFP.  These  business  drivers  correspond  to  the  business  goals 
that  are  motivating  the  system  acquisition  and,  in  turn,  will  drive  the  specification  of  the 
system’s  quality  attributes.  Examples  of  business  drivers  include 

•  time  to  deploy 

•  ability  to  accommodate  operational  upgrades 

•  integration  of  new  subsystems 

•  compatibility  with  Defense  Information  Infrastructure  (DII)  Common  Operating 
Environment  (COE)  or  Command,  Control,  Communications,  Computer  Intelligence, 
Surveillance,  Reconnaissance  (C4ISR)  framework 

•  reduced  maintenance  (i.e.,  modifiability) 

Along  with  the  preliminary  business  case,  the  acquirer  documents  some  preliminary  quality 
attributes  and  associated  scenarios. 


2)  The  supplier  uses  the  preliminary  business  drivers  and  preliminary  quality  attributes  to 
identify  the  system  quality  attributes  to  be  evaluated.  The  acquirer  should  promote  or  be 
amenable  to  having  the  supplier  augment  the  business  drivers  to  reflect  internal  (or  derived) 
business  drivers  that  will  enhance  the  system.  The  system  quality  attributes  become 
contractually  binding  in  accordance  with  the  SOW  requirements  and  the  contract 
deliverables.  This  activity  occurs  during  the  ATAM  step  of  identifying,  prioritizing,  and 
refining  the  most  important  quality  attribute  goals  in  a  utility  tree.  The  figure  illustrates  this 
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step  as  the  activities  associated  with  identification  of  key  quality  attributes.  Note  that  the 
acquirer  and  supplier  jointly  conduct  this  step. 

3)  Next,  the  supplier,  in  conjunction  with  the  acquirer,  refines  and  develops  scenarios  and 
associated  test  cases  to  help  evaluate  the  identified  key  quality  attributes. 

4)  An  analysis  readiness  review  (ARR)  is  typically  used  to  determine  the  timing  of  the 
ATAM  evaluation,  and  ensures  that  the  evaluation  does  not  occur  until  the  supplier  is  actually 
ready.  The  readiness  review  may  be  conducted  to  determine  the  maturity  of  the  software 
architecture,  the  sufficiency  of  the  architecture  documentation,  and  the  adequacy  of  the 
architecture  evaluation  plan.  These  responsibilities  must  be  clearly  delineated  in  the  RFP. 

Our  example,  shown  in  Figure  4,  positions  evaluations  at  the  beginning  and  at  the  end  of  the 
software  architecture  development  phase.  Evaluations  can  occur  at  other  points  in  system 
development  as  well.  The  flow  coming  out  of  the  last  evaluation  checkpoint  shows  that  the 
intent  is  to  allow  corrective  action  to  take  place  as  necessary. 

After  the  initial  evaluation,  the  software  architecture  is  typically  placed  under  configuration 
control,  and  all  subsequent  system  builds  conform  to  the  software  architecture. 

Architecture  status  can  be  presented  at  regularly  scheduled  acquisition  reviews.  Event-driven 
reviews  are  held  to  resolve  issues.  Formal  contract  adjustments  may  be  required  as  both  the 
acquirer’s  and  supplier’s  understanding  of  system  requirements  and  relevant  engineering 
tradeoffs  evolve. 


4.2  RFP/Contract  Language  for  Acquisition  Example 

For  our  example  and  with  the  approach  discussed  above,  we  describe  language  that  typically 
is  included  in  Section  C  of  the  RFR 


Statement  of  Work  (SOW) 

1)  Early  after  contract  award,  the  SOW  requires  the  supplier  to  deliver  a  plan  for  conducting 
the  ATAM  evaluation.  The  plan  must  describe 

•  what — specific  tasks  to  be  accomplished 

•  where — locations  and  facilities  at  which  the  ATAM  will  be  conducted 

•  when — when  the  architecture  evaluations  are  to  take  place  and  a  schedule  of  all 
related  tasks  and  events,  including  the  architecture  readiness  review 

•  who — roles  and  responsibilities  of  stakeholders,  including  the  acquirer 

•  how — specific  procedure  descriptions  for  each  task,  detailing  the  inputs,  steps, 
expected  outcomes 
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These  features  are  not  unique  to  planning  architecture  evaluations;  they  are 
characteristic  of  good  planning  practices  and  planning  artifacts. 

2)  Next  in  Section  C,  the  supplier  is  asked  to  use  the  preliminary  business  drivers  and 
preliminary  quality  attributes  and  scenarios  to  refine  and  develop  additional  business  drivers 
and  quality  attributes  for  the  system.  The  result  is  documented  in  a  quality  attribute  tree  that 
categorized  and  prioritizes  the  attributes. 

3)  With  the  quality  attribute  tree,  the  supplier  is  asked  to  detail,  refine,  and  develop  scenarios 
to  be  used  to  evaluate  the  architecture. 

4)  The  supplier  then  begins  initial  design  of  the  system  and  associated  software  architecture. 

5)  Once  the  initial  design  is  formulated  and  documented,  an  analysis  readiness  review  is  held 
to  ensure  that  the  supplier  has  sufficient  capability  and  capacity  to  conduct  the  ATAM 
evaluation. 

6)  The  supplier  and  acquirer  use  the  initial  design  and  jointly  conduct  a  software  architecture 
evaluation,  in  this  case  using  the  ATAM. 

Although  Figure  4  shows  two  evaluations,  one  after  the  initial  design  and  one  at  the  end  of 
the  system  implementation,  this  technical  note  and  the  RFP/contract  language  provided  in  the 
appendices  focus  on  the  initial  evaluation. 


Product  Requirements 

The  product  requirements  are  described  in  the  system  specification  that  is  also  part  of  Section 
C.  These  requirements  contain  the  preliminary  business  drivers  and  preliminary  quality 
attributes  for  the  system.  From  the  SOW  above,  the  supplier  must  abstract,  or  augment,  these 
requirements  to  identify  the  business  drivers  at  a  level  that  is  more  expressive. 

We  note  again  that  the  SOW  language  and  system  specification  language  here  focus  on  the 
use  of  an  ATAM  evaluation  and  must  be  set  in  the  context  of  the  entire  acquisition.  There  are 
other  requirements  that  are  needed  for  the  acquisition. 
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5  Summary 


In  this  note,  we  have  discussed  how  a  software  architecture  evaluation,  such  as  an  ATAM 
evaluation,  might  be  used  to  reduce  risk  in  a  system  acquisition.  We  have  given  examples  of 
RFP  language  that  may  be  adapted  for  a  particular  acquisition.  Future  technical  notes  will 
include  RFP  language  covering  additional  source  selection  documents,  Sections  M  and  L. 

The  SEI  is  collaborating  with  several  acquisition  organizations  on  the  use  of  the  ATAM  to 
help  them  adopt  and  integrate  the  architecture  evaluations  into  their  own  organizations.  This 
includes  identifying  the  appropriate  language  to  include  in  an  RFP  to  make  an  architecture 
evaluation  an  integral  part  of  evaluating  proposals  as  well  as  system  developments. 
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Appendix  A 


Architecture  Tradeoff  Analysis  Method 
Statement  of  Work  (SOW) 


ATAM 

The  supplier,  in  conjunction  with  the  acquirer,  shall  plan  and  conduct  the  ATAM  evaluation 
for  the  software  architecture  of  the  system  being  acquired.  The  supplier  shall  follow  the 
principles  and  steps  specified  in  the  Technical  Report:  Kazman,  R.  ATAM:  Method  for 
Architecture  Evaluation  (CMU/SEI-2000-TR-004).  Pittsburgh,  PA:  Software  Engineering 
Institute,  Carnegie  Mellon  University,  2000.  See  Attachment  A  of  this  SOW.  The  supplier 
shall  analyze  and  use  the  Business  Drivers  provided  in  the  system  specification  for  the 
system  being  acquired.  These  Business  Drivers  and  other  desired  quality  attributes  of  the 
system  are  specified  as  part  of  the  System  Specification  of  this  RFP/Contract. 


ATAM  Planning 

The  supplier  shall  develop  a  plan  to  conduct  the  ATAM  evaluations  for  the  system  being 
developed.  The  plan  shall  document  the 

•  overall  approach  used  to  conduct  the  ATAM  evaluations 

•  specific  tasks  to  be  accomplished,  including  an  analysis  readiness  review  (ARR) 

•  locations  and  facilities  at  which  the  ATAM  evaluations  will  be  conducted 

•  schedule  of  all  related  tasks  and  events,  including  the  analysis  readiness  review 

•  roles  and  responsibilities  of  stakeholders,  including  of  the  acquirer 

•  specific  procedure  descriptions  for  each  task,  detailing  the  inputs,  steps,  and  expected 
outcomes 


The  supplier  shall  include  all  the  ATAM  steps  in  this  planning  activity.  (See  Attachment  A  to 
this  SOW.) 
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Generation  of  Quality  Attribute  Utility  Tree 


The  supplier,  in  conjunction  with  the  acquirer,  and  using  the  Business  Drivers  and  other 
quality  attributes  specified  in  the  system  specification,  shall  identify,  prioritize,  and  refine  the 
system’s  important  quality  attributes.  (These  quality  attributes  constitute  system  “utility” 
[e.g.,  performance,  reliability,  security,  modifiability]  as  defined  in  the  system  specification.) 
At  the  supplier’s  request,  the  acquirer  will  make  available  representative  personnel 
(stakeholders)  to  assist  the  supplier  in  understanding  requirements  for  generating  the  utility 
tree. 


Scenario  Development 

In  conjunction  with  the  acquirer,  the  supplier  shall  develop  and  document  initial  scenarios,  to 
be  used  in  architecture  tradeoff  analysis  (hereafter  referred  to  as  analysis).  Scenarios 
developed  will  be  sufficiently  defined  and  documented  to  be  employed  in  the  analysis. 
Scenarios  will  be  developed  to  fully  exercise  the  system  and  software  architecture  to 
determine  the  degree  to  which  the  specified  system  requirements,  including  quality  attributes, 
are  or  will  be  satisfied. 

Following  the  design  of  these  scenarios,  the  supplier  and  the  acquirer  shall  conduct  a 
technical  exchange  meeting,  assessing  the  scenarios  to  determine  whether  they  are  sufficient 
to  be  utilized  in  the  ATAM  process.  Weaknesses  or  deficiencies  in  these  scenarios  found 
during  this  review  will  be  entered  into  the  supplier’s  corrective  action  system  and  resolved  by 
the  supplier  prior  to  conducting  the  ATAM  evaluation. 


System/Software  Architecture  Development 

The  supplier  shall  develop  the  software  architecture  in  conjunction  with  the  development  of 
the  system  and  system  architecture.  The  software  architecture  will  be  designed  to  satisfy  or 
support  the  Business  Drivers,  and  system  requirements,  including  all  quality  attributes, 
specified  in  the  System  Specification.  This  engineering  effort  may  utilize  prototyping 
methods  or  implementing  selected  components  to  enable  tradeoffs  and  mitigate  high-risk 
areas  in  order  to  ensure  the  software  architecture  design  will  satisfy  system  requirements.  The 
supplier  will  conduct  technical  exchange  meetings  with  the  acquirer,  as  appropriate,  during 
system  and  software  architecture  development  to  convey  status  of  the  development  effort, 
identify  risks,  and  mutually  resolve  issues. 

During  this  development,  the  supplier  will  document  the  software  architecture.  The 
documentation  must  be  sufficient  to  support  the  ATAM. 
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The  software  architecture  design  will  be  completed  and  the  ATAM  evaluation  conducted 

prior  to  the  supplier  proceeding  with  full  system  implementation.  Completion  of  this 

architecture  definition  and  evaluation  effort  will  provide  an  initial  design  of  the  software 

architecture  in  which 

•  software  components  in  each  architecture  have  been  identified  and  interfaces  to  those 
software  components  have  been  described 

•  architecture  relationships  among  the  software  components  have  been  described,  such  as 
data  flow,  process  synchronization,  usage,  and  resource  sharing 

•  allocation  of  functionality  to  software  components  has  been  resolved  and  allocation 
documented 

•  key  architecture  conflicts  among  software  components  have  been  identified  and  resolved 

•  conflicts  with  satisfaction  of  quality  attributes  have  been  identified  and  resolved 

•  documentation  of  the  architecture  is  sufficient  for  evaluation 

•  rationale  for  the  design  decisions  present  in  the  architecture  has  been  described 


Analysis  Readiness  Review  (ARR) 

As  part  of  conducting  the  ATAM  evaluation  and  following  the  completion  of  the  software 
architecture  design,  the  supplier  shall  plan  and  jointly  conduct  with  the  acquirer  an  analysis 
readiness  review  (ARR)  to  determine  that  the  software  architecture  design  is  sufficiently 
complete  and  properly  documented  to  enable  the  ATAM  evaluation  to  be  conducted  and  to 
identify  any  issues  in  the  design.  The  ARR  will  include  an  assessment  of  the  software 
architecture  documentation  and  the  supplier’s  plan  for  the  evaluation.  Weaknesses  or 
deficiencies — in  the  architecture,  its  documentation,  or  the  ATAM  plan — found  during  this 
review  will  be  entered  into  the  supplier’s  corrective  action  system  and  resolved  by  the 
supplier  prior  to  the  ATAM  evaluation. 


Conduct  the  ATAM  Evaluation 

After  resolving  issues  identified  during  the  ARR,  the  supplier  shall  finalize  the  ATAM  plan 
and  jointly  conduct  the  ATAM  evaluation  with  the  acquirer.  The  supplier  shall  follow  the 
principles  and  steps  specified  in  the  Technical  Report:  Kazman,  R.  ATAM:  Method  for 
Architecture  Evaluation  (CMU/SEI-2000-TR-004).  Pittsburgh,  PA:  Software  Engineering 
Institute,  Carnegie  Mellon  University,  2000.  See  Attachment  A  of  this  SOW.  The  analysis  is 
intended  to  identify  tradeoff  points  in  software  architecture  (i.e.,  those  elements  that  affect 
multiple  quality  attributes  of  the  system  and  software).  The  analysis  will  be  used  to  determine 
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the  extent  to  which  the  software  architecture  satisfies  or  supports  the  contractual 
requirements  and  whether 

•  the  models  of  the  attributes  need  to  be  refined  and  more  analyses  conducted 

•  the  architecture  needs  to  be  refined 

•  the  models  need  to  be  changed  to  reflect  these  refinements 

•  the  system  requirements  and  contract  requirements  need  to  be  changed 


The  analysis  will  use  the  supplier-generated  use  cases  and  scenarios  developed  as  stated 
above.  The  supplier  will  be  responsible  for  recording  the  results  of  the  analysis  (e.g., 
sensitivity  points,  tradeoff  points,  risks,  and  issues).  Any  risks  or  issues  will  be  entered  into 
the  supplier’s  tracking/corrective-action  system  with  plans  for  resolution  or  mitigation.  All 
issues  are  to  be  resolved  by  the  supplier  prior  to  implementation  of  the  system.  Following  the 
analysis  and  resolution  of  issues,  and  planning  the  mitigation  of  risks  identified  during  the 
analysis,  the  supplier  shall  place  the  software  architecture  under  configuration  control,  using 
the  supplier’s  configuration  management  system. 

Any  changes  or  impacts  to  supplier  work  efforts  resulting  from  this  analysis  will  be  entered 
into  the  supplier’s  effort-tracking  system  and  formally  communicated  to  the  acquirer  in 
accordance  with  standard  contractual  procedures  and  specific  terms  of  the  contract 
agreement. 


Incremental  System  Development 

Following  the  analysis  and  making  of  any  changes  to  the  system  or  software  architecture 
requirements  resulting  from  and  recommended  by  the  analysis,  the  supplier  shall 
incrementally  implement  builds  to  the  system  according  to  the  schedule  of  builds  specified  in 
the  System  Engineering  Plan.  Changes  to  the  software  architecture  during  system 
implementation  shall  be  controlled  using  the  supplier’s  configuration  management  system, 
including  documented  rationale  for  the  changes. 


Documentation  of  Engineering  Efforts 

The  engineering  efforts  during  the  development  of  the  software  architecture,  including  all 
evaluations  and  analyses,  will  be  documented  and  will  include  the  rationale  for  both  design 
decisions  and  changes  to  the  baseline  architecture.  Documentation  will  be  sufficient  to 
support  the  architecture  evaluation  method  and  the  tracking  of  changes  to  the  baselined 
software  architecture.  Specific  information  about  the  architectures  must  include,  but  is  not 
limited  to,  module  structure,  component  interfaces,  process  structure,  and  data-flow  structure. 
In  each  structure,  the  view-specific  relationships  among  the  entities  must  be  documented.  For 
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the  module  structure,  relationship  information  includes,  but  is  not  limited  to,  the  unique 
information  that  is  encapsulated  in  each  module.  For  the  process  structure,  the  relationship 
information  includes,  but  is  not  limited  to,  synchronization  and  concurrency  relationships. 
For  the  data-flow  structure,  relationship  information  includes,  but  is  not  limited  to,  a  high- 
level  description  of  the  data  that  is  produced,  stored,  consumed,  or  transformed. 


Reviews  and  Technical  interchange  Meetings 

The  supplier  shall  address  the  progress  of  the  software  architecture  development  effort  at 
normally  scheduled  acquisition  reviews,  and,  as  required,  resolve  software  architecture- 
related  issues.  In  addition,  the  supplier  shall  conduct  technical  interchange  meetings  with  the 
acquirer  at  specified  times. 


Adjustment  to  Contractual  Requirements 

As  a  result  of  the  design  effort,  or  as  the  understanding  of  the  system  and  the  software 
architecture  improves,  adjustments  to  the  contractual  requirements  may  be  made  upon  mutual 
agreement  between  the  supplier  and  the  acquirer.  Adjustments  to  the  contractual  requirements 
will  be  made  through  established  contractual  means. 
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Appendix  B 


Product  Requirements 


Specification  Section:  Requirements 

Program  scope:  a  new  radar  system  acquisition  that  includes  developing  a  production-quality 
software  architecture.  The  following  are  Business  Goals  and  initial  Quality  Attributes  for  the 
program: 


Business  Goals 

The  program  expects  significant  enhancements  to  follow  the  first  system  delivery. 
Affordability  of  these  enhancements  is  critical. 

The  program  expects  that  the  system  can 

•  keep  pace  with  changing  requirements 

•  keep  pace  with  changing  computer  platforms 

•  keep  pace  with  commercial  technology.  Commercial  components  will  be  integrated  in  the 
software  where  appropriate  to  keep  pace  with  commercially  available  technology. 

The  program  expects  that  the  system  will  resist  intrusion  into  and  operation  of  the  system  by 
non-authorized  sources  while  still  providing  its  services  to  legitimate  users. 

The  program  expects  the  system  will  ensure  operation  under  maximum  load  conditions  (i.e., 
tracking  specified  number  of  targets  simultaneously). 


Quality  Attributes 

The  following  quality  attributes  for  the  system  are  derived  from  the  Business  Drivers  for  the 
program. 

Run-Time  Requirements:  The  software  architecture  will  ensure  achievement  of  system 
functions  and  the  quality  requirements  of  modifiability,  reliability,  and  security,  as  described 
in  this  specification.  See  glossary  of  this  document  for  definitions  of  these  quality  attributes 
as  used  in  this  acquisition. 

Non-Run-Time  Requirements:  The  software  will  be  compliant  with  DII  COE. 
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Potential  Evaluation  Scenarios 

The  software  architecture  will  also  ensure  achievement  of  the  system  quality  attributes  shown 
in  the  following  potential  evaluation  scenarios: 


Quality  Attribute 

Potential  Evaluation  Scenarios 

Modifiability 

The  system  is  expected  to  have  changes  in  the 
following  areas: 

•  output  data  formats 

•  radar  signal  inputs 

•  radar  search  control  signals 

•  incorporation  of  COTS  for  DBMS 
functionality 

Reliability 

Target  information  will  be  simulated  to  overload  the 
system  with  50  targets  simultaneously. 

Security 

Attempts  to  compromise  the  system  will  be  made  by 
an  unauthorized  operator  and  through  the 
communication  system  connected  to  an  external 
client. 

Specification  Section:  Qualification 

Analysis  of  the  system  quality  attributes  will  use  the  ATAM  described  in  the  Technical 
Report:  Kazman,  R.  ATAM:  Method  for  Architecture  Evaluation  (CMU/SEI-2000-TR-004). 
Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon  University,  2000,  using  the 
change  scenarios  described  herein. 
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Glossary 


Functionality 


Modifiability 


Performance 


Reliability 


Scenario 


Security 


Software 

Architecture 


The  ability  of  the  system  to  do  the  work  for  which  it  was  intended. 


The  extent  to  which  the  system  can  be  changed  quickly  and  cost- 
effectively. 


The  responsiveness  of  the  system  -  the  time  required  to  respond  to 
stimuli  (events),  or  the  number  of  events  processed  in  some  interval  of 
time. 


A  measure  of  the  proportion  of  time  the  system  is  up  and  running. 


A  brief  description  of  a  stakeholder’s  interaction  with  a  system;  how  a 
system  behaves  or  interacts  with  the  stakeholders  to  accomplish  desired 
objectives  or  stated  requirements. 


A  measure  of  the  system’s  ability  to  resist  unauthorized  attempts  at 
usage  and  denial  of  service,  while  still  providing  its  services  to 
legitimate  users. 


System  and  the  structure  or  structures  of  the  system,  which  comprise 
software  components,  the  externally  visible  properties  of  those 
components,  and  the  relationships  among  them. 
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Feedback  and  Contact 


Comments  or  suggestions  about  this  document  or  the  series  of  technical  notes  on  architecture 
evaluation  in  the  Department  of  Defense  are  welcome.  We  want  this  series  to  be  responsive 
to  the  needs  of  DoD  and  government  personnel.  To  that  end,  comments  concerning  this 
technical  note,  inclusion  of  other  topics,  or  any  other  issues  or  concerns  will  be  of  great  value 
in  continuing  this  series.  Comments  or  suggestions  should  be  sent  to 

Linda  Northrop,  Director 
Product  Line  Systems  Program 
lmn@sei.cmu.edu 


Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 
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architecture  evaluations  in  a  system  acquisition  and  describes  the  contractual  elements  that  are  needed  to 
accommodate  architecture  evaluations  in  an  acquisition.  The  note  then  provides  an  example  of  contractual 
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